Information processing apparatus and subject information acquisition method

ABSTRACT

A point-of-interest information acquisition section of an information processing apparatus acquires, by using a polarization image captured from a viewpoint of an imaging apparatus, an incident plane relative to the viewpoint at a point of interest ‘a’ on a subject. A viewpoint control section determines a direction of movement of the imaging apparatus in such a manner as to suitably acquire an angle formed with the incident plane and presents the direction to a user. When the post-movement viewpoint of an imaging apparatus is determined to be appropriate, an incident plane at the point of interest ‘a’ is acquired by using the captured polarization image, and a line of intersection with the incident plane is assumed to be a normal vector n of the point of interest ‘a.’

TECHNICAL FIELD

The present invention relates to an information processing apparatus anda subject information acquisition method for acquiring states of asubject by using a captured image.

BACKGROUND ART

There have been available techniques to recognize a certain object in asubject space and detect its position and posture by analyzing acaptured image, finding application in a wide range of fields includingelectronic content such as games, object and space modeling, andmonitoring cameras. Various techniques are under study as techniques foracquiring three-dimensional information such as position and posturefrom two-dimensional information of a subject in the captured image, andthere has been proposed, for example, a technique for obtaining a normalon an object surface by using a polarization image (refer, for example,to NPL 1 and NPL 2).

[CITATION LIST] [NON PATENT LITERATURES]

[NPL 1] Jeremy Riviere, et al. “Polarization imaging reflectometry inthe wild,” Technical Report 2016/8, Department of Computing. ImperialCollege London, ISSN 1469-4174, May 2016.

[NPL 2] Zhaopeng Cui, et al. “Polarimetric Multi-View Stereo,”Proceedings of the IEEE Conference on Computer Vision and PatternRecognition, 2017.

[Summary] [Technical Problems]

Image analysis based on polarized light generally focuses on a change inluminance relative to a polarization azimuth, thus offering highrobustness to surrounding brightness, presence or absence of featurepoints on a subject surface, and so on. Meanwhile, observed lightincludes specular reflection and diffuse reflection, two kinds of lightthat differ in a manner of reflection, possibly resulting in degradedcomputation accuracy depending on an aptitude of a model.

For this reason, application scenes are limited such as using imageanalysis in combination with information regarding a distance to thesubject obtained separately by a stereo camera or infrared radiation assupplemental means and using image analysis for a material whose mannerof reflection is known. Although techniques called inverse renderinghave been proposed that calculate observed light by assuming unknownparameters such as material and normal and derive such parameters insuch a manner as to achieve a match with actual observation results,these techniques involve a high processing load, making themdisadvantageous under a situation where high response speed, inparticular, is required.

The present invention has been devised in light of the foregoingproblems, and it is an object of the present invention to provide atechnique for readily acquiring subject information by using apolarization image.

Solution to Problems

A mode of the present invention relates to an information processingapparatus. This information processing apparatus includes a capturedimage acquisition section adapted to acquire data of polarization imagesin a plurality of azimuths captured by an imaging apparatus fromdifferent viewpoints, an imaging apparatus information acquisitionsection adapted to acquire information regarding a position and postureof the imaging apparatus as viewpoint information, a point-of-interestinformation acquisition section adapted to acquire, by usingpolarization luminance of a pixel of interest representing a point ofinterest on a subject, an incident plane of observed light at the pointof interest for each of the viewpoints first, acquire point-of-intereststate information in a world coordinate system by integrating theincident planes on the basis of a positional relation between theviewpoints, and output the point-of-interest state information, and aviewpoint control section adapted to determine, on the basis of theincident plane acquired relative to a first viewpoint, a direction inwhich to move the viewpoint and determine a post-movement viewpoint whena given condition is met relative to the incident plane as a secondviewpoint for which to derive the incident plane next.

Another mode of the present invention relates to a subject informationacquisition method. This subject information acquisition method, used byan information processing apparatus, includes a step of acquiring dataof polarization images in a plurality of azimuths captured by an imagingapparatus from different viewpoints, a step of acquiring informationregarding a position and posture of the imaging apparatus as viewpointinformation, a step of acquiring, by using polarization luminance of apixel of interest representing a point of interest on a subject, anincident plane of observed light at the point of interest for each ofthe viewpoints first and acquiring point-of-interest state informationin a world coordinate system by integrating the incident planes on thebasis of a positional relation between the viewpoints, a step ofdetermining, on the basis of the incident plane acquired relative to afirst viewpoint, a direction in which to move the viewpoint anddetermining a post-movement viewpoint when a given condition is metrelative to the incident plane as a second viewpoint for which to derivethe incident plane next, and a step of outputting the point-of-intereststate information.

It should be noted that any combinations of the above components andconversions of expressions of the present invention between a method, anapparatus, and the like are also effective as modes of the presentinvention.

Advantageous Effect of Invention

According to the present invention, it is possible to readily acquireinformation regarding a position and posture of a subject by using apolarization image.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram illustrating a configuration example of aninformation processing system in the present embodiment.

FIG. 2 is a diagram schematically illustrating a capturing environmentof polarization images used in the present embodiment.

FIG. 3 depicts diagrams each illustrating change in luminance relativeto a polarization azimuth used in the present embodiment.

FIG. 4 is a diagram comparing change in a degree of polarizationrelative to a zenith angle of a normal vector between specularreflection and diffuse reflection.

FIG. 5 is a diagram illustrating an example of a structure of an imagingdevice having a polarizer layer that can be introduced into an imagingapparatus in the present embodiment.

FIG. 6 is a diagram illustrating an internal circuit configuration of aninformation processing apparatus in the present embodiment.

FIG. 7 is a diagram illustrating a functional block configuration of theinformation processing apparatus in the present embodiment.

FIG. 8 is a flowchart illustrating a processing procedure for theinformation processing apparatus to acquire subject state information byusing a polarization image in the present embodiment.

FIG. 9 is a diagram schematically illustrating a positional relationbetween a viewpoint of the imaging apparatus, a point of interest on asubject, and a pixel of interest on a captured image in the presentembodiment.

FIG. 10 is a diagram schematically illustrating a manner in which anormal at the point of interest is acquired from incident planescorresponding to a plurality of viewpoints in the present embodiment.

FIG. 11 depicts diagrams each illustrating an example of a screendisplayed on a display apparatus via an output data generation sectionfor a viewpoint control section to guide a viewpoint of the imagingapparatus in the present embodiment.

FIG. 12 is a flowchart illustrating a processing procedure for apoint-of-interest information acquisition section to acquire subjectstate information in S22 in FIG. 8.

FIG. 13 is a diagram schematically illustrating a manner in which aline-of-sight vector from a first viewpoint is projected onto an imageplane of a second viewpoint in the present embodiment.

FIG. 14 depicts diagrams for describing a technique for evaluating areliability level of the normal vector by use of a relation between thezenith angle and the degree of polarization in S40 in FIG. 12.

FIG. 15 is a diagram for describing another example of the technique forevaluating the reliability level of the normal vector by use of therelation between the zenith angle and the degree of polarization in S40in FIG. 12.

FIG. 16 is a diagram for describing adequacy of use of the degree ofpolarization for evaluating the reliability level of the normal vectorin the present embodiment.

FIG. 17 is a diagram for describing a technique for identifying anincident plane on the basis of the change in the degree of polarizationrelative to the zenith angle θ in the present embodiment.

DESCRIPTION OF EMBODIMENT

FIG. 1 illustrates a configuration example of an information processingsystem in the present embodiment. This information processing systemincludes an imaging apparatus 12, an information processing apparatus10, and a display apparatus 16. The imaging apparatus 12 captures animage of a subject 8. The information processing apparatus 10 acquiresdata of the captured image and performs a given information processingtask. The display apparatus 16 outputs a result of the informationprocessing. The information processing system may further include aninput apparatus that accepts operation on the information processingapparatus 10 from a user. The information processing apparatus 10 mayfurther be capable of communicating with an external apparatus such as aserver by connecting to a network such as the Internet.

The information processing apparatus 10, the imaging apparatus 12, andthe display apparatus 16 may be connected by cables or wirelessly bywireless local area network (LAN) or the like. Also, two or more of theinformation processing apparatus 10, the imaging apparatus 12, and thedisplay apparatus 16 may be combined into an integral apparatus. Forexample, an information processing system may be realized by using acamera or a mobile terminal having these apparatuses. In the case ofusing a camera, an electronic finder may be used as the displayapparatus 16. Alternatively, a head-mounted display that is worn on theuser's head and displays an image in front of the user's eyes may beused as the display apparatus 16, and the imaging apparatus 12 may beprovided on the head-mounted display in such a manner as to capture animage corresponding to a user's line of sight. In any case, theinformation processing apparatus 10, the imaging apparatus 12, and thedisplay apparatus 16 are not limited to those illustrated in appearanceand shape.

In such a system, the information processing apparatus 10 acquirespolarization image data captured by the imaging apparatus 12 andidentifies at least either a normal at a point of interest on thesubject 8 or a position in a three-dimensional space. Then, as a resultthereof, the information processing apparatus 10 generates image andsound data and outputs the data to the display apparatus 16. There maybe one or a plurality of points of interest on the subject 8. Forexample, it is possible to identify the shape of the subject 8 bysetting points of interest with density equal to or higher than a givenvalue on the subject 8 and obtaining each position.

If, on top of that, the normal at one of the points of interest isobtained at a given frequency, a change in posture of the subject 8 canbe acquired. Naturally, the change in posture of the subject 8 and adestination thereof can be acquired by continuously acquiring thepositions of all the points of interest. Hereinafter, the normals at thepoints of interest and the positions thereof and the shape and postureof the subject may be collectively referred to as “subject stateinformation.” Contents of data to be output from the informationprocessing apparatus 10 on the basis of subject state informationacquired by using a polarization image are not limited.

For example, data that represents acquired state information itself maybe output, or an environmental map of a subject space may be generatedby integrating these pieces of data and output. Alternatively,information processing may be performed separately by using such stateinformation, followed by output of the result thereof by image or sound.For example, an electronic game or an arbitrary information processingtask may be progressed by using a given target included in the subject 8as a controller of the game and recognizing a motion thereof as useroperation.

Alternatively, a virtual world may be represented by replacing a subjectspace including the subject 8 with a virtual object, or a virtual objectinteracting with the subject 8 may be drawn on a captured image. Avirtual world drawn within a field of view corresponding to the user'sline of sight may be displayed on the head-mounted display by modelingthe real world as a virtual object. Subject state information acquiredby using a polarization image may be stored temporarily in a storageapparatus or the like for use for other information processing task at alater time.

FIG. 2 schematically illustrates a capturing environment of polarizationimages used in the present embodiment. The imaging apparatus 12 capturesan image of a space including a subject 72 via a linear polarizer plate70. In more details, the imaging apparatus 12 observes, of reflectedlight that includes a specular reflection component obtained as a resultof causing light emitted from a light source 74 to be reflected by thesubject 72 and a diffuse reflection component obtained as a result ofcausing the light emitted from the light source 74 to be scatteredinside the subject 72, polarized light that has passed through thelinear polarizer plate 70.

Here, the linear polarizer plate 70 transmits, of reflected light thatreaches the imaging apparatus 12 from the subject 72, only linearpolarized light oscillating in a certain direction (referred to as a“polarization azimuth”). The rotation of the linear polarizer plate 70about an axis vertical to its surface allows for the polarizationazimuth to be set up in an arbitrary direction. Assuming that light thatreaches the imaging apparatus 12 is non-polarized light, observedluminance is constant even if the linear polarizer plate 70 is rotated.Meanwhile, when partially polarized, common reflected light experienceschange in luminance observed in the polarization azimuth.

Light observed as a silhouette of a point of interest ‘a’ on the subject72 is reflected light inside an incident plane 76 including a normalvector n of the subject 72 at that position. It is known that so-calleds polarized light that oscillates in a direction vertical to theincident plane 76 is predominant in specular reflection and thatso-called p polarized light that oscillates in a direction parallel tothe incident plane 76 is predominant in diffuse reflection. Also, aratio between s polarized light and p polarized light depends upon anangle θ (referred to as a “zenith angle”) formed between the normalvector n and a light beam observed on the incident plane 76.

For this reason, an extent of polarization, i.e., a degree ofpolarization, and a polarization phase vary depending upon the incidentplane 76, determined by the relation between the viewpoint of theimaging apparatus 12 and the normal vector n of the point of interest‘a,’ the zenith angle θ, and the ratio between specular reflection anddiffuse reflection. In other words, it is possible to estimate thenormal vector n as seen from the imaging apparatus 12 by rotating thelinear polarizer plate 70 and acquiring a change in luminance relativeto a polarization azimuth after assuming the ratio between specularreflection and diffuse reflection. It should be noted that, in thedescription given hereinafter, the term “obtain an incident plane”refers to obtaining an incident plane angle on a captured image plane orin a three-dimensional space. Also, the term “viewpoint” of the imagingapparatus 12 can include a lens direction in addition to a lens centerposition of the imaging apparatus 12.

FIG. 3 illustrates change in luminance I relative to the polarizationazimuth 1. The graph on the upper side in FIG. 3 illustrates a casewhere specular reflection is predominant, and the graph on the lowerside in FIG. 3 illustrates a case where diffuse reflection ispredominant, and each is in a shape of a sine wave with a 180° period.Meanwhile, a polarization azimuth vs when the luminance I in specularreflection has a maximum value Imax differs by 90° from a polarizationazimuth ψd when the luminance I in diffuse reflection has the maximumvalue Imax. This is attributable, as described above, to the fact thatthe s polarized light is predominant in specular reflection and that thep polarized light is predominant in diffuse reflection.

Considering the fact that the s polarized light is oscillation verticalto the incident plane and that the p polarized light is oscillationparallel to the incident plane, the polarization azimuth (ψs-90°) thatprovides the lowest luminance in specular reflection or the polarizationazimuth ψd that provides the highest luminance in diffuse reflectionrepresents the angle of the incident plane. The normal vector n isalways included in the incident plane. Therefore, the angle in questionrepresents the angle of the vector obtained by projecting the normalvector n onto the captured image plane. This angle is commonly referredto as an azimuth angle of the normal vector n. Obtaining the zenithangle on the incident plane in addition to the azimuth angle in questionallows for a normal vector to be uniquely determined in athree-dimensional space as seen from the imaging apparatus 12.Hereinafter, the polarization azimuth that provides the maximumluminance of observed polarized light will be referred to as a phaseangle v. The change in the luminance I illustrated in FIG. 3 can beexpressed by the following formula by using the phase angle ψ.

[Math. 1]

$\begin{matrix}{I = {\frac{I_{\max} + I_{\min}}{2} + {\frac{I_{\max} - I_{\min}}{2}{\cos\left( {{2\phi} - {2\psi}} \right)}}}} & \left( {{Formula}\mspace{14mu} 1} \right)\end{matrix}$

I_(max), I_(min), and ψ can be obtained by approximating the luminanceobserved for the plurality of polarization azimuths Φ resulting from therotation of the linear polarizer plate 70 to a format of Formula 1 byusing the least squares method or the like. Of these, I_(max) andI_(min) can be used to obtain a degree of polarization ρ by using thefollowing formula.

[Math. 2]

$\begin{matrix}{\rho = \frac{I_{\max} - I_{\min}}{I_{\max} + I_{\min}}} & \left( {{Formula}\mspace{14mu} 2} \right)\end{matrix}$

FIG. 4 compares change in the degree of polarization relative to thezenith angle of the normal vector between specular reflection anddiffuse reflection. In the case of specular reflection illustrated inthe graph on the upper side, the degree of polarization takes on amaximum value of up to 1.0. In contrast, in the case of diffusereflection illustrated in the graph on the lower side, the degree ofpolarization takes on a maximum value of 0.4 or so. The angle of theincident plane relative to the phase angle ψ differs by 90° dependingupon which of specular reflection and diffuse reflection is predominant.That is, even if the phase angle ψ is obtained by expressing the changein luminance relative to the polarization azimuth as in Formula 1, a 90°uncertainty occurs on the incident plane.

For this reason, the normal vector is identified in the presentembodiment on the basis of a specular reflection model by setting athreshold Th_(ρ) for the degree of polarization, selecting a pointhaving a greater degree of polarization, and specifying this point as apoint of interest. That is, a behavior of the luminance of polarizedlight having a degree of polarization equal to or higher than thethreshold Th_(ρ) relative to the polarization azimuth is approximated tothe format of Formula 1, and the polarization azimuth (Ψ-90°) thatprovides the lowest luminance in question is used as the angle of theincident plane. It should be noted that a degree of polarization ρ_(s)of specular reflected light can be expressed by the following formula asa function of the zenith angle θ and a refractive index η of thesubject.

[Math. 3]

$\begin{matrix}{{\rho_{s}\left( {\eta,\theta} \right)} = \frac{2\sin^{2}{\theta cos\theta}\sqrt{\eta^{2} - {\sin^{2}\theta}}}{\eta^{2} - {\sin^{2}\theta} - {\eta^{2}\sin^{2}\theta} + {2\sin^{4}\theta}}} & \left( {{Formula}\mspace{14mu} 3} \right)\end{matrix}$

The illustrated degree of polarization typically represents the casewhere the refractive index η is 1.6. A common artifact has a refractiveindex that does not change significantly and remains approximatelybetween 1.5 and 1.6. Therefore, it is safe to assume that the thresholdTh_(ρ) is constant irrespective of the subject. It should be noted,however, that the threshold Th_(ρ) may be set in a rigorous manner tosuit the material of the subject. Also, in the present embodiment, thechange in the degree of polarization of specular reflection relative tothe zenith angle will be used to evaluate the reliability level of theestimated normal vector as will be described later. In this case, it isalso possible to make an evaluation with similar computations regardlessof the subject by use of the fact that a tendency of the degree ofpolarization relative to the refractive index η does not changesignificantly. Meanwhile, the degree of polarization may be obtained ina rigorous manner depending on the subject material for use forevaluation purposes.

It is possible to derive the normal vector angle inside the incidentplane, i.e., the zenith angle θ from a degree of polarization ρ by usingthe relation of Formula 3. However, the azimuth angle and the zenithangle of the normal vector obtained in this manner are merely withreference to the viewpoint of the imaging apparatus 12. Also, positioncoordinates of a point of interest in a world coordinate system cannotbe acquired only from the information in question. For this reason,analysis using a polarization image is commonly often used assupplemental means to interpolate a distance value from the imagingapparatus 12 to the subject, identify a change in posture of the subjectwhose position is known, or the like.

Meanwhile, in the present embodiment, the viewpoint of the imagingapparatus 12 is varied freely, and of those viewpoints, at leastpolarization images from two thereof are analyzed, thus acquiring aplurality of incident planes for the same point of interest. Then,pieces of incident plane information, each for one of the viewpoints inquestion, are integrated on the basis of the positional relation betweenthe viewpoints, thus acquiring state information at the point ofinterest in the world coordinate system. Specifically, because a normalvector is included in any incident plane, the normal vector is obtainedin the world coordinate system from a line of intersection of at leasttwo incident planes.

Here, the reliability level of the acquired normal vector is evaluatedon the basis of the relation between the zenith angle and the degree ofpolarization determined by the positional relation between the normalvector and the viewpoint. Further, position coordinates of a point ofinterest in the world coordinate system are acquired by acquiring inwhich direction the point of interest is located from the two viewpointsin question. As a result, state information of the subject in the worldcoordinate system can be acquired by using only polarization images.

It should be noted that means of observing polarization luminance is notlimited to a linear polarizer plate in the present embodiment. Forexample, a polarizer layer may be provided as part of an imaging devicestructure. FIG. 5 illustrates an example of a structure of an imagingdevice having a polarizer layer that can be introduced into the imagingapparatus 12 in the present embodiment. It should be noted that FIG. 5schematically illustrates a functional structure of a cross-section ofthe device and that detailed structures such as an interlayer insulatingfilm and interconnects are not depicted. An imaging device 110 includesa microlens layer 112, a wire grid type polarizer layer 114, a colorfilter layer 116, and a photo-detection layer 118.

The wire grid type polarizer layer 114 includes polarizers having aplurality of linear conductor members arranged in a striped pattern atspacings smaller than a wavelength of incident light. When lightconcentrated by the microlens layer 112 enters the wire grid typepolarizer layer 114, polarization components whose azimuths are parallelto lines of the polarizer are reflected, thus allowing only polarizationcomponents vertical to the lines to pass. A polarization image isacquired by detecting the polarization components that have passed withthe photo-detection layer 118. The photo-detection layer 118 has asemiconductor device structure such as that of a common charge coupleddevice (CCD) image sensor or a common complementary metal oxidesemiconductor (CMOS) image sensor.

The wire grid type polarizer layer 114 includes an arrangement ofpolarizers that provide different azimuths of transmitting polarizedlight from one charge readout unit to another, i.e., from one pixel toanother, or in larger units in the photo-detection layer 118. Apolarizer arrangement 120, visible as one sees the wire grid typepolarizer layer 114 from above, is illustrated on the right in FIG. 5.The lines shaded in FIG. 5 are conductors (wires) included in thepolarizers. It should be noted that each of rectangles with dotted linesrepresents a polarizer region in one direction and that the dotted linesthemselves are not actually formed.

In the example illustrated, polarizers in four directions are arrangedin four regions 122 a, 122 b, 122 c, and 122 d, in two rows by twocolumns. In the figure, the polarizers at opposite angles have theirtransmission directions orthogonal to each other, and the polarizersadjacent to each other have their transmission directions that differ by45°. That is, polarizers are provided in four directions, 45° apart fromeach other. These polarizers serve as a substitute for the linearpolarizer plate 70, making it possible to acquire polarizationinformation regarding four azimuths, 45° apart from each other, inregions of the photo-detection layer 118 provided underneath, eachcorresponding to one of the four regions 122 a, 122 b, 122 c, and 122 d.By further arranging a given number of such polarizers vertically andhorizontally and connecting peripheral circuitry for controlling chargereadout timings, it is possible to realize an image sensor thatsimultaneously acquires polarization information regarding four azimuthsas two-dimensional data.

In the imaging device 110 illustrated in FIG. 5, the color filter layer116 is provided between the wire grid type polarizer layer 114 and thephoto-detection layer 118. The color filter layer 116 includes, forexample, an arrangement of respective filters through which red light,green light, and blue light pass in association with the respectivepixels. This provides polarization information by color in accordancewith the combination of the direction of the polarizer in the wire gridtype polarizer layer 114 and the filter color in the color filter layer116 that is located under the wire grid type polarizer layer 114. Thatis, polarization information in the same azimuth and for the same coloris obtained discretely on the image plane. As a result, a polarizationimage for each azimuth and for each color is obtained by interpolatingthe polarization information as appropriate.

Also, it is possible to reproduce a non-polarization color image byperforming computations on polarization images of the same color. Imageacquisition techniques using a wire grid type polarizer are alsodisclosed, for example, in JP 2012-80065A. It should be noted, however,that polarization luminance images are basically used in the presentembodiment. Therefore, if color images are not required in other usages,the color filter layer 116 may be omitted. Also, the polarizers are notlimited to a wire grid type, and linear dichroic polarizers or othertype of polarizers may also be used.

In the case where azimuth dependence of polarization luminance isapproximated to the format of Formula 1, polarization luminance of atleast three azimuths is required for the same point of interest.According to the illustrated imaging device, polarization luminance offour azimuths can be acquired at the same time for approximately thesame point on the subject, thus meeting this condition. There is only asmaller degree of freedom than the linear polarizer plate 70, anddepending on the case, an approximation formula may include large error.In the present embodiment, however, points whose degree of polarizationis equal to or higher than the threshold Th_(ρ) are processed asdescribed above. That is, only those points with a large differencebetween I_(max) and I_(min) are subject to computations, thus making itunlikely for an approximation formula to include error even in the caseof polarization of limited azimuths and making it possible to obtain thephase angle ψ, and by extension, subject state information, with highaccuracy.

FIG. 6 illustrates an internal circuit configuration of the informationprocessing apparatus 10. The information processing apparatus 10includes a central processing unit (CPU) 23, a graphics processing unit(GPU) 24, and a main memory 26. These sections are connected to eachother via a bus 30. An input/output interface 28 is further connected tothe bus 30. A communication section 32, a storage section 34, an outputsection 36, an input section 38, and a recording medium driving section40 are connected to the input/output interface 28. The communicationsection 32 includes a peripheral equipment interface such as universalserial bus (USB) or IEEE (Institute of Electrical and ElectronicEngineers) 1394 and a wired or wireless LAN network interface. Thestorage section 34 includes a hard disk drive or a non-volatile memory.The output section 36 outputs data to the display apparatus 16. Theinput section 38 receives data input from the imaging apparatus 12 andan input apparatus that is not illustrated. The recording medium drivingsection 40 drives a removable recording medium such as a magnetic disk,an optical disc, or a semiconductor memory.

The CPU 23 controls the information processing apparatus 10 as a wholeby executing an operating system stored in the storage section 34. TheCPU 23 also executes various programs read out from the removablerecording medium and loaded into the main memory 26 or downloaded viathe communication section 32. The GPU 24 has a geometry engine functionand a rendering processor function, performing a drawing process inaccordance with a drawing instruction from the CPU 23 and storingdisplay image data in a frame buffer that is not illustrated. Then, theGPU 24 converts the display image stored in the frame buffer into avideo signal, outputting the signal to the output section 36. The mainmemory 26 includes a random access memory (RAM) and stores programs anddata required for processing.

FIG. 7 illustrates a functional block configuration of the informationprocessing apparatus 10 of the present embodiment. Each element recitedas a functional block for performing various processing tasks in FIG. 7can be configured by hardware such as various circuits including the CPU23, the GPU 24, and the main memory 26 illustrated in FIG. 6 and can berealized by software such as programs loaded into the main memory 26from the recording medium driven by the recording medium driving section40 and the storage section 34. Therefore, it is understood by a personskilled in the art that these functional blocks can be realized invarious ways including hardware alone, software alone, and a combinationthereof, and the functional blocks are not limited to any one of them.

The information processing apparatus 10 includes a captured imageacquisition section 50, an image data storage section 52, an imagingapparatus information acquisition section 53, a subject informationacquisition section 54, and an output data generation section 56. Thecaptured image acquisition section 50 acquires captured image data fromthe imaging apparatus 12. The image data storage section 52 storesacquired image data. The imaging apparatus information acquisitionsection 53 acquires position and posture information of the imagingapparatus 12. The subject information acquisition section 54 acquiresinformation regarding the normal and shape of the subject on the basisof the captured image. The output data generation section 56 generatesdata to be output on the basis of subject state information.

The captured image acquisition section 50 is realized by the inputsection 38, the CPU 23, and the like in FIG. 6 and acquires capturedimage data including polarization images, captured from a plurality ofviewpoints, from the imaging apparatus 12. At this time, the capturedimage acquisition section 50 acquires, as the polarization images,images with polarization luminance in at least three azimuths perviewpoint. In the case where an image sensor having a polarizer layerincluding polarizers oriented in a plurality of directions as describedabove is introduced in particular, video data whose image frames arepolarization images including information in a plurality of azimuths maybe acquired. At this time, video data that is captured while theviewpoint is continuously moved at the same time may be acquired.

Alternatively, still image data captured separately from two or moreviewpoints may be acquired. Still alternatively, a plurality ofstationary cameras having different viewpoints may be used as theimaging apparatus 12, so that still images or video data captured byeach camera is acquired. Depending on the purpose of informationprocessing and the details of image analysis as in a case where acaptured image is displayed on the display apparatus 16, the capturedimage acquisition section 50 may further acquire common color capturedimage data. The captured image acquisition section 50 stores acquiredcaptured image data in the image data storage section 52.

It should be noted that in the case where a single image frame includespolarization information in a plurality of azimuths, the captured imageacquisition section 50 generates polarization images in a plurality ofazimuths by separating and interpolating, as appropriate, pixel valuesfor each polarization azimuth first and then stores the polarizationimages in the image data storage section 52. The imaging apparatusinformation acquisition section 53 is realized by the input section 38,the CPU 23, and the like in FIG. 6 and acquires a position and postureof the imaging apparatus 12 in a real space at least when thepolarization images are captured. Typically, the imaging apparatusinformation acquisition section 53 derives, in given time steps, theposition and posture of the imaging apparatus 12 on the basis ofmeasured values such as acceleration and angular velocity measured bymotion sensors incorporated in the imaging apparatus 12.

Alternatively, the position and posture of the imaging apparatus 12 maybe derived by an image analysis technique such as simultaneouslocalization and mapping (SLAM) on the basis of captured images acquiredby the captured image acquisition section 50. These techniques arewidely known. Therefore, the description thereof is omitted. The subjectinformation acquisition section 54 is realized by the CPU 23, the GPU24, and the like in FIG. 6 and acquires a normal vector and positioncoordinates of a point of interest on the subject in the worldcoordinate system by using the polarization image data stored in theimage data storage section 52 and position and posture information ofthe imaging apparatus 12 acquired by the imaging apparatus informationacquisition section 53. The subject information acquisition section 54may acquire the shape and posture of the subject by treating a set ofpoints of interest as a subject's surface.

In more details, the subject information acquisition section 54 includesa viewpoint control section 60 and a point-of-interest informationacquisition section 64. The viewpoint control section 60 performscontrol such that polarization images are captured from suitableviewpoints by using position and posture information of the imagingapparatus 12. In a mode where the user moves the imaging apparatus 12 byholding or wearing it, the viewpoint control section 60 determines apolarization image captured from a certain viewpoint as a reference andnotifies the user of the direction in which the viewpoint moves insubsequent image captures. The notification may be made by displaying animage on the display apparatus 16 or producing a sound via the outputdata generation section 56.

In the present embodiment, a normal and position coordinates are derivedin the world coordinate system by using incident planes obtained for aplurality of viewpoints and a line of sight to a point of interest asdescribed above. At this time, the farther the other viewpoint is fromthe incident plane of one of the viewpoints, the less likely it is for acomputation result to include error. Therefore, the viewpoint controlsection 60 guides the viewpoint in the direction away from the incidentplane of the former viewpoint, and by extension, in the direction ofbringing the incident plane closer to 90° on condition that the point ofinterest remains within the field of view. Then, a polarization imagecaptured when an appropriate viewpoint is acquired is used as a targetimage to be analyzed. At this time, the user may capture a still imagein response to a shutter timing notified by the viewpoint controlsection 60. Alternatively, the viewpoint control section 60 may extractan image frame from an appropriate viewpoint from among a series ofpieces of video image data.

It should be noted that the present embodiment is not limited to a modewhere the user moves the imaging apparatus 12, and a plurality ofimaging apparatuses may be fixed in position at an appropriate viewpointas described above. Alternatively, a mechanism whose position andposture can be controlled by the viewpoint control section 60 may beprovided in the imaging apparatus 12. It should be noted, however, thatthe present embodiment permits acquisition of subject state informationby using images captured from a small number of viewpoints such as twoviewpoints. Therefore, even the mode where the user moves the imagingapparatus 12 does not involve much labor and, moreover, keepsinstallation cost to a minimum. Hereinafter, a viewpoint from which areference polarization image is captured will be referred to as a “firstviewpoint,” and a viewpoint having an appropriate positional relationwith the first viewpoint in terms of analysis will be referred to as a“second viewpoint.” It should be noted that there may be only one secondviewpoint or two or more second viewpoints.

The point-of-interest information acquisition section 64 sets a pixel ofinterest for a polarization image captured from the first viewpoint andacquires an incident plane on the basis of azimuth dependence ofpolarization luminance in the pixel of interest. Here, the term “pixelof interest” refers to a pixel obtained by projecting a point ofinterest on a subject in a three-dimensional space onto an image plane.That is, this process is nothing but setting a point of interest on thesubject and determining an incident plane at the point of interestrelative to the first viewpoint. As pixels of interest, pixels whosedegree of polarization is equal to or higher than a threshold areselected as described above. The acquired incident plane is used by theviewpoint control section 60 to control second viewpoints.

The point-of-interest information acquisition section 64 furtheracquires an incident plane at a pixel representing the same point ofinterest on the subject for a polarization image captured from thesecond viewpoint. Here, in the polarization image captured from thesecond viewpoint, the pixel position representing the same point ofinterest as set by using the first viewpoint image is unknown unless,for example, there is an obvious feature point. For this reason, thepoint-of-interest information acquisition section 64 projects theline-of-sight vector from the first viewpoint to the point of interestonto the image plane of the second viewpoint, acquiring normal vectorsof the subject on a straight line thereof. Then, the reliability levelof each normal vector is evaluated by use of the relation between thezenith angle and the degree of polarization, thus identifying a pixelrepresenting the same point of interest.

This makes it possible to identify a corresponding pixel of interest onthe image plane imaged from the second viewpoint and acquire a normalvector of the point of interest at the same time. It should be notedthat these processes may be performed in parallel for a plurality ofpoints of interest. The point-of-interest information acquisitionsection 64 may further evaluate the normal vectors acquired in thismanner according to a given criterion and assign a reliability level. Inthe case where the reliability level is lower than a given threshold,the normal vector in question may be excluded from the output result.Alternatively, each normal vector may be output in association with areliability level.

Further, the point-of-interest information acquisition section 64extends, as necessary, the line of sight from each viewpoint to thepixel of interest on the image plane, thus acquiring an intersectionthereof as position coordinates of the point of interest. Also in thiscase, data to be output is screened on the basis of the reliabilitylevel acquired for the corresponding normal vector or is associated witha reliability level and output.

The output data generation section 56 is realized by the CPU 23, the GPU24, the output section 36, and the like in FIG. 6 and outputs subjectstate information acquired by the subject information acquisitionsection 54 or generates and outputs data representing results of a giveninformation processing task performed on the basis of the subject stateinformation. For example, in the case where a real object existing in asubject space is replaced with a virtual object or a virtual object isdrawn that interacts with a real object, the output data generationsection 56 creates, as an initial process of such a usage, anenvironmental map obtained by modeling a real space and outputs the map.At this time, the map may be output to a storage apparatus such as themain memory 26.

Alternatively, the output data generation section 56 may perform a giveninformation processing task such as game by using output data of thesubject information acquisition section 54 and output a display image orsound data representing the result thereof to the display apparatus 16.At this time, the output data generation section 56 may use, for exampleand as appropriate, natural light captured images stored in the imagedata storage section 52. It is understood by a person skilled in the artthat various information processing tasks can be realized by usingnormals and shapes of real objects.

A description will be given next of operation of the informationprocessing apparatus 10 that can be realized by the configurationdescribed above. FIG. 8 is a flowchart illustrating a processingprocedure for the information processing apparatus 10 to acquire subjectstate information by using a polarization image. This flowchart assumesthat the user moves the viewpoint by moving the imaging apparatus 12.Also, the procedure progresses in parallel with the acquisition ofinformation regarding the position and posture thereof by the imagingapparatus information acquisition section 53 and the acquisition ofcaptured image data by the captured image acquisition section 50 afterthe information processing apparatus 10 has established communicationwith the imaging apparatus 12.

The viewpoint control section 60 of the subject information acquisitionsection 54 acquires image data from a first viewpoint first (S10). Atthis time, a still image captured from a certain viewpoint may be usedin an ‘as-is’ fashion as a first viewpoint image. Alternatively, of avideo captured continuously from varying viewpoints, an image frameextracted by the viewpoint control section 60 may be selected as a firstviewpoint image.

Next, the point-of-interest information acquisition section 64 sets apixel of interest for the first viewpoint polarization image (S12).Specifically, the point-of-interest information acquisition section 64approximates a change in polarization luminance to Formula 1 for eachpixel of the polarization image first and selects those pixels whosedegree of polarization obtained from Formula 2 is equal to or higherthan the threshold Th_(ρ) as pixels of interest. It should be noted thatthe purpose is not to select all such pixels as pixels of interest.Instead, only a silhouette region of a target object may be selected.Alternatively, only those pixels at given intervals of the region may beselected. Still alternatively, only a single pixel may be selected as apixel of interest. In the case where a single pixel of interest is set,the pixel that provides the highest degree of polarization may beselected rather than a decision based on a threshold.

Further, the point-of-interest information acquisition section 64obtains an incident plane (reference incident plane) relative to thefirst viewpoint from the phase angle ψ obtained by approximation toFormula 1 for each pixel of interest (S14). As a result of selection ofpixels whose degree of polarization is large as pixels of interest,ψ-90° is acquired as an incident plane angle by applying a specularreflection model. By using the first viewpoint information in the worldcoordinate system, i.e., position and posture information of the imagingapparatus 12, it is possible to define a plane having ψ-90° relative tothe viewpoint in question in the world coordinate system.

Next, the viewpoint control section 60 guides the viewpoint of theimaging apparatus 12 in such a manner as to acquire a second viewpointin an appropriate manner (S16). Here, the term “appropriate viewpoint”refers, as described above, to a viewpoint where the point of intereston the subject represented by the pixel of interest set in S12 is withinthe field of view and where the angle formed between the incident planesis sufficient. It should be noted, however, that the point of interestand the position of a silhouette thereof are unknown at this point intime. Therefore, it is impossible to precisely acquire incident planesrelative to the second viewpoint. For this reason, the angle formedbetween the incident planes is, for example, approximated by the angleformed by an optical axis relative to the incident plane of the firstviewpoint.

In the case where a normal vector is obtained from two incident planescorresponding to two viewpoints, it is desirable that the angle formedby the incident planes should be close to 90°. Meanwhile, in the casewhere a normal vector is obtained from incident planes corresponding tothree or more viewpoints, it is desirable that the maximum angle formedby two incident planes should be close to 90° and that other incidentplanes should be acquired by an angle obtained by evenly dividing theangle in question. Therefore, an appropriate threshold for the opticalaxis angle is set in advance to suit the number of viewpoints for acaptured image to be used for analysis. Alternatively, the aptitude maybe simply decided by using the viewpoint positions. In this case, forexample, a viewpoint is determined to be appropriate when the distancefrom the incident plane of the first viewpoint is equal to or higherthan a threshold.

While the viewpoint fails to meet a condition for an appropriateviewpoint (N in S18), the viewpoint control section 60 continues toguide the viewpoint (S16). When the condition is met (Y in S18), theviewpoint control section 60 selects a polarization image captured atthat time as a second viewpoint image (S20). It should be noted that inthe case where images captured from three or more viewpoints are used,the processes in S16 to S20 are repeated. The number of viewpoints forpolarization images used for processing is set properly on the basis ofrequired accuracy, processing capability of the information processingapparatus 10, time permitted for processing, and the like.

Next, the point-of-interest information acquisition section 64 acquiresan incident plane at the same point of interest by using thepolarization image captured from the second viewpoint first and thenderives a normal vector of the point of interest by obtaining a line ofintersection with the incident plane of the first viewpoint. Further,the point-of-interest information acquisition section 64 derivesposition coordinates of the point of interest by obtaining anintersection of line-of-sight vectors, each from one of the viewpointsto the point of interest (S22). In this process, the normal vector isevaluated in terms of reliability level in accordance with a conditionat the time of derivation of the normal vector, consistency withsurrounding results, and the like, followed by screening as appropriateand assignment of a reliability level.

The point-of-interest information acquisition section 64 supplies, tothe output data generation section 56, a normal vector and positioncoordinates in the world coordinate system in association with areliability level as appropriate for each point of interest on thesubject. The output data generation section 56 generates datarepresenting results themselves or output data resulting from processessuch as games and image drawing by using the data and outputs the data(S24). In the case where an image processing task is performed by usingsubject state information, it is possible to screen values used for theinformation processing task or adjust an impact of the reliabilitylevel, a weight, on processing results.

FIG. 9 schematically illustrates a positional relation between aviewpoint of the imaging apparatus 12, a point of interest on a subject,and a pixel of interest on a captured image. According to the commoncentral projection type imaging apparatus 12 assumed in the presentembodiment, a silhouette of the point of interest ‘a’ on the subject 72is projected onto an intersection b between a straight line L connectinga viewpoint 84 of the imaging apparatus 12 and the point of interest ‘a’and a plane 80 of the captured image. Here, the plane 80 of the capturedimage is vertical to the optical axis at a position a focal distance faway from the viewpoint 84 toward an optical axis O, and the sizethereof is determined by a viewing angle of the imaging apparatus 12. Inthe present embodiment, we assume that internal parameters of theimaging apparatus 12 such as the focal distance f and the viewing angleare acquired in advance.

As the imaging apparatus information acquisition section 53 acquires theposition and posture of the imaging apparatus 12, the positioncoordinates of the viewpoint 84 and an orientation of the optical axis Obecome known. Therefore, the plane 80 of the captured image can bedefined in the world coordinate system. Considering the projectionrelation described above, even if the position coordinates of the pointof interest ‘a’ on the subject are unknown, the straight line L wherethe point of interest ‘a’ can exist is uniquely determined by giving theposition coordinates of the point on the plane 80 of the captured imageonto which the point of interest ‘a’ is projected, i.e., the pixel ofinterest b. Hereinafter, a vector from the viewpoint 84 of the imagingapparatus 12 toward the point of interest ‘a’ will be referred to as aline-of-sight vector.

Meanwhile, the phase angle ψ and the degree of polarization ρ areobtained for each pixel on the plane 80 from Formulas 1 and 2 on thebasis of azimuth dependence of polarization luminance. The selection ofa point whose degree of polarization ρ is equal to or higher than agiven threshold as the pixel of interest b allows for ψ-90°, the angleof the incident plane 85, to be obtained from the phase angle ψ byapplying a specular reflection model. The incident plane 85 in questionis a plane that includes the line-of-sight vector L and the normalvector n of the point of interest ‘a.’ Because the plane 80 of thecaptured image has been acquired in the global coordinate system, theincident plane 85 is also defined in the world coordinate system.

FIG. 10 schematically illustrates a manner in which a normal at a pointof interest is acquired from incident planes corresponding to aplurality of viewpoints. Here, the viewpoints of imaging apparatuses 12a and 12 b are assumed to be first and second viewpoints, respectively.First, at the first viewpoint, an incident plane 86 a acquired for thepoint of interest ‘a’ on the subject 72 includes a line-of-sight vectorL1 toward the point of interest ‘a’ and the normal vector n of the pointof interest ‘a.’ Here, if an incident plane 86 b is obtained in the samemanner after moving the imaging apparatus 12 to the second viewpoint,the normal vector n agrees with a line of intersection between theincident plane 86 a and the incident plane 86 b. Also, an intersectionbetween a line-of-sight vector L2 at this time and the line-of-sightvector L1 from the first viewpoint agrees with the point of interest‘a.’

Here, the larger the angle between the line-of-sight vector L1 and theline-of-sight vector L2 corresponding to two viewpoints, and moreprecisely, the closer the angle formed by the line-of-sight vector L2 ofthe second viewpoint and the incident plane 86 a of the first viewpointto 90°, the better noise immunity, and as a result, the higher theaccuracy of the normal vector n and the position coordinates of thepoint of interest ‘a.’ For this reason, the viewpoint control section 60presents, to the user, information regarding a proper direction in whichto move the viewpoint and a direction in which to orient the lens asdescribed above.

FIG. 11 depicts diagrams each illustrating an example of a screendisplayed on the display apparatus 16 via the output data generationsection 56 for the viewpoint control section 60 to guide the viewpointof the imaging apparatus 12. The rectangle depicted in each of (a), (b),(c), and (d) represents a screen displayed when a capture is made withthe first viewpoint, and a captured image from the first viewpoint isillustrated as a base. In this example, an image of a spherical subject90 is captured. The same figure also depicts that a pixel of interest 92is set on a silhouette of a subject 90. However, this may not beactually displayed.

The viewpoint control section 60 guides the viewpoint such that thepixel of interest 92 set on the first viewpoint captured image remainswithin the field of view and that the angle formed between the incidentplanes approaches 90° as described above. In the example of (a), acircle 96 surrounding the pixel of interest 92 and arrows 94 a and 94 bstarting from the circle 96 and pointing leftward and rightward,respectively, are displayed in a superimposed manner on the capturedimage. The directions of the arrows point away from the incident planeof the first viewpoint and do not necessarily point leftward orrightward as illustrated. With the arrows 94 a and 94 b displayed in asuperimposed manner, for example, guidance, in text or voice, isprovided saying “Move the camera in one of the directions indicated bythe arrows while keeping the circle near the center of the screen.”

While the user moves the imaging apparatus 12 in either direction inaccordance with the guidance, the appearance of the subject spacecaptured by the imaging apparatus 12, the arrows displayed in asuperimposed manner, and the like are continuously displayed. Forexample, the circle 96 indicating the pixel of interest may be made tolook as if it is affixed on the subject 90 by moving the circle 96 inthe direction opposite to the direction of movement of the viewpoint ofthe imaging apparatus 12. The movement of the viewpoint can beidentified on the basis of position and posture information of theimaging apparatus 12 acquired by the imaging apparatus informationacquisition section 53. As the viewpoint approaches an appropriate one,a message to this effect may be made clear, for example, by shorteningthe arrow in the corresponding direction.

In the case of (b), a feature point such as a pattern 98 actuallyexisting on the subject is used. That is, once the pixel of interest 92is set, a feature point that is sized to be visible by the user and hasa sufficient difference in color from surrounding areas is detected inthe vicinity thereof and in the direction of movement toward theappropriate viewpoint. A common edge detection technique or a commonpattern matching technique can be used for extraction. Then, theviewpoint is guided to move toward the detected feature point, forexample, by providing text or voice guidance saying “Move the camerasuch that the yellow star pattern is placed at the front center.”

An arrow may be displayed near the feature point in a superimposedmanner to indicate where the feature point is located. While the usermoves the imaging apparatus 12 in accordance with the guidance, theappearance of the subject space captured by the imaging apparatus 12 iscontinuously displayed. This allows the user to adjust the viewpoint byconfirming the actual angle.

In the case of (c), the subject is marked on the spot, and the markingis used as a feature point. That is, once the pixel of interest 92 isset, graphics is displayed in a superimposed manner indicating theposition a given length away in the direction of movement toward theappropriate viewpoint, and moreover, a text or voice instruction isgiven to mark the subject surface at that position. In the exampleillustrated, a speech bubble 100 indicating a text saying “Apply asticker here” is displayed in a superimposed manner. In order totemporarily mark the subject, thin objects that do not affect thesubject shape much such as pieces of paper or stickers whose color,shape, and size are easy to distinguish from surrounding areas are madeavailable separately in advance. Depending upon the subject material, amarking may be drawn directly on the subject with an erasable pen.

Also in this mode, text or voice guidance is provided, at the time ofapplication of the marking, saying “Move the camera such that thesticker is placed at the front center,” thus guiding the viewpoint tomove toward the marking. While the user moves the imaging apparatus 12,the appearance of the subject space captured by the imaging apparatus 12is continuously displayed. It should be noted that, although aninstruction is given to apply a sticker to a single location in theexample illustrated, the two directions of movement to the appropriateviewpoint may be both specified, thus allowing the user to choose themore convenient of the two for marking.

In the example of (d), a light beam shining device 102 such as laserpointer that can be controlled by the viewpoint control section 60 ismade available in the real space in advance, and a light beam is shoneonto the position a given length away in the direction of movementtoward the appropriate viewpoint. This allows a shining pattern 104 tobe formed on the subject. Moreover, for example, text or voice guidanceis provided saying “Move the camera such that the shining pattern isplaced at the front center,” thus guiding the viewpoint to move towardthe shining pattern 104. In this case, the viewpoint control section 60switches on or off the shining device and adjusts the shining position.Alternatively, the shining device may be provided on the side of theimaging apparatus 12 and controlled by the viewpoint control section 60.

When a polarization image is captured from the appropriate viewpoint asa result of viewpoint guidance described above, the point-of-interestinformation acquisition section 64 obtains an incident plane for thepixel of interest representing the same point of interest on the subjectin polarization images captured from a plurality of viewpoints on thebasis of azimuth dependence of polarization luminance as illustrated inFIG. 10. Here, in the case where the position of the point of intereston the subject is unknown, a silhouette of the point of interest inquestion in the second viewpoint captured image, i.e., where the pixelof interest is located, is unknown. For this reason, thepoint-of-interest information acquisition section 64 repeatedly derivesa normal vector, thus searching for a correct pixel of interest on thesecond viewpoint captured image.

FIG. 12 is a flowchart illustrating a processing procedure for thepoint-of-interest information acquisition section 64 to acquire subjectstate information in S22 in FIG. 8. First, the point-of-interestinformation acquisition section 64 projects the line-of-sight vector L1from the first viewpoint to the point of interest onto the plane of thesecond viewpoint captured image. As illustrated in FIG. 9, the pixel ofinterest b is set in the plane 80 of the captured image of the firstviewpoint 84, thus allowing a line-of-sight vector L thereof to beuniquely determined in a three-dimensional space. Meanwhile, an imageplane 80 b of the second viewpoint is obtained from the position andposture of the imaging apparatus 12 b at that time as illustrated inFIG. 10. As a result, the line-of-sight vector L1 from the firstviewpoint can be represented as a straight line on the image plane 80 bof the second viewpoint.

FIG. 13 schematically illustrates a manner in which the line-of-sightvector L1 from the first viewpoint is projected onto the image plane ofthe second viewpoint. As described above, the point of interest on thesubject is located at any one of the positions on the line-of-sightvector L1 from the first viewpoint. For this reason, thepoint-of-interest information acquisition section 64 searches for acorrect pixel of interest on the straight line obtained by projectingthe line-of-sight vector L1 onto the image plane 80 b of the secondviewpoint.

The same figure illustrates, by angular directional arrows, the mannerin which phase angles ψ1, ψ2, ψ3, and ψ4 are obtained from azimuthdependence of polarization luminance for pixels p1, p2, p3, and p4 onthe straight line representing the line-of-sight vector L1. Basically,the reliability level of a normal vector obtained in the case wherethese pixels are assumed to be pixels of interest is evaluated, thusdetermining the pixel that provides the most probable normal vector as apixel of interest and the normal vector in question as a normal vectorof the point of interest.

Referring back to FIG. 12, the point-of-interest information acquisitionsection 64 sets, as a target, a certain pixel on the straight lineobtained by projecting the line-of-sight vector L1 (S32) and obtains thedegree of polarization ρ by using Formulas 1 and 2 on the basis of thepolarization luminance at that position (S34). In S12 in FIG. 8, a pixelwhere specular reflection is predominant is selected when the pixel ofinterest is set for the first viewpoint image. The degree ofpolarization is dependent upon the zenith angle. Therefore, in the casewhere the second viewpoint is varied primarily in an azimuthaldirection, it is probable that the degree of polarization is also equalto or higher than the given value also at the pixel of interest in thesecond viewpoint image. Therefore, no more calculations are performedfor those pixels whose degree of polarization ρ is smaller than thethreshold by determining that these pixels are not corresponding pixels(N in S36).

In this case, a next pixel is set as a target on the projected straightline, and the degree of polarization is estimated similarly (S32 toS36). It should be noted that the term “next pixel” may be an adjacentpixel on the projected straight line or a pixel located at a givenspacing as illustrated in FIG. 13. Also, the threshold used as adecision criterion in S36 may be the same as or different from the oneused when a pixel of interest is set for the first viewpoint image inS12 in FIG. 8.

If the degree of polarization is equal to or higher than the threshold(Y in S36), the phase angle ψ is obtained by Formula 1, followed bycalculation of the incident plane angle with a specular reflection modeland obtaining of a line of intersection with the incident plane acquiredrelative to the first viewpoint, thus estimating the normal vector n asillustrated in FIG. 10 (S38). It should be noted, however, that we donot know whether the incident plane of the point of interest has beenacquired properly. For this reason, the reliability level of theacquired normal vector is evaluated on the basis of the relation betweenthe degree of polarization and the zenith angle (S40).

Specifically, the degree of polarization theoretically acquired from thezenith angle determined from an estimated normal vector is compared withthe degree of polarization actually obtained from Formulas 1 and 2, andif it is concluded that there is no inconsistency, the estimated normalvector is determined to be correct (N in S40). In this case, thepoint-of-interest information acquisition section 64 calculates areliability level for the normal vector in question first (S42) andrecords them in association with each other in the main memory or thelike (S44). Here, the term “reliability level” may be a functionobtained in the decision process of S40 where the smaller the differencebetween the theoretical degree of polarization and the actual degree ofpolarization, the larger the value. Alternatively, a new reliabilitylevel may be calculated from a point of view different from a degree ofpolarization.

For example, in the case where a plurality of points of interest are seton the same subject, the reliability level is calculated on the basis ofdispersion of the normal vectors acquired for the points of interestwithin a given range. Alternatively, the refractive index η of thesubject is obtained by substituting the zenith angle θ determined by thenormal vector and the degree of polarization ρ determined by Formula 2into Formula 3, after which the reliability level is calculated on thebasis of dispersion thereof. In either case, a possible approach wouldbe to derive an increase in dispersion resulting from a newly acquirednormal vector and calculate a reliability level with a function wherethe larger the increase, the smaller the value.

Meanwhile, if it is concluded in S40 that there is inconsistency such asa difference greater than the threshold between the theoretical degreeof polarization and the actually acquired degree of polarization (Y inS40), a next pixel is set as a target on the projected straight line andsimilar processes are repeated by deciding that the estimated normalvector is not correct (S32 to S40). It should be noted that in the casewhere there is inconsistency in degree of polarization for all thepixels on the projected straight line such as in the case where thesecond viewpoint is not appropriate, the process may be terminated atthat point in time (not depicted). According to the above processes, itis possible to simultaneously acquire a true pixel of interest on theimage plane of the second viewpoint, the true pixel of interestcorresponding to the point of interest on the subject, and a normalvector of the point of interest in question regardless of the positionof the point of interest in the world coordinate system.

It should be noted that, in the flowchart illustrated in FIG. 12, if acertain pixel provides a highly reliable normal vector for a point ofinterest, the process is terminated after recording the normal vector.This makes it possible to save time required for processing. Meanwhile,the most reliable normal vector may be selected and recorded byperforming the processes from S34 to S40 or to S42 on all the targetpixels on the straight line obtained by projecting the line-of-sightvector L1 onto the image plane.

In this case, of the pixels sampled at given intervals on the straightline obtained by projecting the line-of-sight vector L1, the mostreliable pixel or that with a reliability level equal to or higher thanthe threshold may be extracted, followed by sampling of the pixelswithin a given range around that location with more density forreevaluation of the reliability level. The number of times the processesare repeated by varying the density of pixels to be sampled may befixed, or the number of times the probability of acquiring a higherreliability level than a previous sampling becomes equal to or higherthan a threshold may be determined adaptively through statisticalprocessing of a reliability level distribution. Also in this case, themost reliable normal vector is selected through processing and recorded.

Also in the description given above, of the lines of intersection of theincident planes from two viewpoints, a highly reliable line is selectedas a normal vector. As described above, a correct pixel of interest inthe second viewpoint image is simultaneously determined at this time. Itis possible, by taking advantage of this, to identify the secondline-of-sight vector L2 that is directed from the second viewpointtoward the point of interest on the subject through the pixel ofinterest in question. For this reason, the position coordinates of thepoint of interest ‘a’ on the subject in the world coordinate system maybe further obtained and recorded in the main memory or the like in S44by obtaining the intersection with the first viewpoint line-of-sightvector L1 as illustrated in FIG. 10. In this mode, the dispersion of theposition coordinates in question may be used as a reliability levelcalculated in S42. Also in this case, it is only necessary to derive theincrease in dispersion resulting from the newly acquired positioncoordinates and calculate the reliability level with a function wherethe larger the increase, the smaller the value.

Also, in the case where captured images from N (where N is a naturalnumber equal to or larger than 2) viewpoints, the illustrated flowchartis repeated N−1 times. In this case, N−1 normal vectors are basicallyacquired for a point of interest. Of these normal vectors, the mostreliable one, a mean vector thereof, a vector estimated by statisticalprocessing, or the like is used as a final normal vector.

It should be noted, however, that a normal vector may not be acquired inS40 because of inconsistency in degree of polarization for all thepixels on the straight line obtained by projecting the line-of-sightvector L1. For this reason, this condition may be taken advantage of tocalculate the reliability level in S42. That is, a ratio of the numberof normal vectors actually acquired to the number of normal vectors N−1to be acquired is used as a reliability level. It should be noted thatonly one of the several reliability levels described so far may beselected, or the plurality thereof may be combined for calculation froma multilateral point of view.

FIG. 14 depicts diagrams for describing a technique for evaluating thereliability level of the normal vector by use of the relation betweenthe zenith angle and the degree of polarization in S40 in FIG. 12.First, a graph 138 a depicted by a dotted line in (a) indicates therelation between the zenith angle and the degree of polarization forspecular reflected light depicted in (a) of FIG. 4. In the case whereall the observed light is specular reflected light, the relationexhibits a change as depicted by the graph 138 a. In reality, a diffusereflection component is often included. Therefore, the relation exhibitsa change obtained by multiplying the degree of polarization of the graph138 a by a given ratio λ(0<λ<1) as depicted by a graph 138 b.

For this reason, the ratio λ is calculated as follows by using a zenithangle θ1 determined by the estimated normal vector and the firstviewpoint line-of-sight vector L1:λ=ρ_(ob_1)/ρ_(th_1)

where ρ_(th_1) is the degree of polarization at the zenith angle θ1 inthe case of presence of only specular reflection, and ρ_(ob_1) is theactual degree of polarization calculated from the first viewpointpolarization image by Formulas 1 and 2. That is, a degree ofpolarization ρ_(ob) of light from the same point of interest on thesubject represented by the graph 138 b can be expressed by the followingfunction by use of the ratio λ and the function ρ_(th)(θ) of the degreeof polarization of only specular reflection:ρ_(ob)(θ)=λ*ρ_(th)(θ)

In (b) of FIG. 14, the graph 138 b of the degree of polarizationρ_(ob)(θ) is depicted again. In the case of a normal vector of a correctpoint of interest, the degree of polarization of light observed at thesecond viewpoint should satisfy the graph 138 b. Letting the zenithangle at the second viewpoint be denoted as θ2, a degree of polarizationρ_(est_2) in that case is theoretically as follows:ρ_(est_2)=λ*ρ_(th)(θ2)Meanwhile, an actual degree of polarization ρ_(ob_2) calculated byFormulas 1 and 2 is separately acquired from the second viewpointpolarization image. If a difference Δρ=|ρ_(ob_2)−ρ_(est_2)| therebetweenis smaller than a given threshold, it is determined that the degree ofpolarization is not inconsistent, and if the difference is equal to orhigher than the threshold, it is determined that the degree ofpolarization is inconsistent.

As described above, the point-of-interest information acquisitionsection 64 may conclude, when a normal vector whose degree ofpolarization is not inconsistent is acquired, that the normal vector inquestion is a true value. Alternatively, the point-of-interestinformation acquisition section 64 may calculate a function where thesmaller Δρ, the larger the value as a reliability level of the normalvector and associate the function with the normal vector in advance, andthen select, at a later time, the most reliable normal vector. Also, theselected normal vector may be associated with the reliability level inquestion and recorded in the main memory or the like.

FIG. 15 is a diagram for describing another example of the technique forevaluating the reliability level of the normal vector by use of therelation between the zenith angle and the degree of polarization in S40in FIG. 12. In this example, a relation in magnitude between the zenithangles θ1 and θ2 of the normal vectors relative to the first and secondviewpoints and a relation in magnitude between the degrees ofpolarization are used. That is, position coordinates (θ1, ρ_(ob_1)) onthe graph 138 b representing the degree of polarization are uniquelydetermined by the zenith angle θ1 obtained at the first viewpoint forthe estimated normal vector. The degree-of-polarization graph 138 b isdivided into two regions, a monotonously increasing region A and amonotonously decreasing region B, with a maximum point as a boundary.

Therefore, it is possible to identify whether the degree of polarizationρ increases or decreases with increase in the zenith angle θ dependingupon in which region the zenith angle θ is located. In the case wherethe position coordinates (θ1, ρ_(ob_1)) are located in the monotonouslyincreasing region A as illustrated, and if the zenith angle θ2 relativeto the second viewpoint is smaller than 01, the degree of polarizationρ_(est_2) therefor should be smaller than the degree of polarizationρ_(ob_1) of the first viewpoint. In contrast, in the case where theactual degree of polarization ρ_(ob_2) calculated by Formulas 1 and 2from the polarization image of the second viewpoint is larger thanρ_(ob_1) as illustrated, it can be said that the reliability level ofthe estimated normal vector is low.

According to this technique, it is possible to evaluate the reliabilitylevel of a normal vector from the relation in magnitude between thezenith angle and the degree of polarization without obtaining a functionof degree of polarization such as that illustrated in FIG. 14 in arigorous manner. Here, whether the degree of polarization increases ordecreases properly with change in the zenith angle may be used as a soleevaluation criterion. Alternatively, an increment or a decrement of thedegree of polarization may be further evaluated. The former qualitativeevaluation is suitable for the case where an observation system has muchnoise, with relatively poorly accurate zenith angle, degree ofpolarization, and other values, allowing for evaluation with the errorin question factored in. The latter quantitative evaluation is suitablefor the case where the observation system has a relatively highlyaccurate zenith angle, degree of polarization, and other values, thusallowing for rigorous evaluation.

It should be noted, however, that the latter technique requires that afunction of the degree of polarization relative to the zenith angle beobtained in consideration of a diffuse reflection component in a mannersimilar to that illustrated in FIG. 14. On top of that, a supposedvariation in degree of polarization Δρ_(est) and an actual variation indegree of polarization Δρ_(ob) when the zenith angle changes from θ1 toθ2 are defined as follows:Δρ_(est)=ρ_(est_2)−ρ_(ob_1)Δρ_(ob)=ρ_(ob_2)−ρ_(ob_1)

Then, a reliability level r is obtained by using a differenced=|Δρ_(ob)−Δρ_(est)| between the two as follows:r=max(0,1−d/c)where 1/c is a fixed coefficient. This reliability level r is a functionthat is 1 when the difference d is 0, linearly decreases when thedifference d is 0<d<c, and is 0 when the difference d is equal to orlarger than c. Therefore, an appropriate value is assigned to c inadvance as a threshold of the difference d when the reliability level is0.

FIG. 16 is a diagram for describing adequacy of using the degree ofpolarization for evaluating a reliability level of the normal vector inthe present embodiment. In the present embodiment, an image is capturedfrom the first viewpoint first, and the second viewpoint is guided inthe direction away from the incident plane at the point of interest ‘a’and such that the point of interest ‘a’ remains within the field ofview. This means that the viewpoint moves along a trajectory close to acircumference of a bottom of a cone 140 having the point of interest ‘a’as its vertex, the zenith angle as its half vertex angle, and the normalvector n as its axis.

That is, even if the viewpoint is moved significantly, the zenith angleθ does not change much. As a result, when polarized light at the sameposition on the subject is observed from the post-movement viewpoint,the degree of polarization thereof does not change much. For thisreason, the selection of a point of interest whose degree ofpolarization is greater than the threshold provides not only anadvantageous effect of making the application of a specular reflectionmodel applicable but also an advantageous effect of permitting efficientextraction of pixels representing the same point of interest even if theviewpoint moves. Also, because the zenith angle θ does not change much,it is highly likely that the degree of polarization thereof remains inthe monotonously increasing region or the monotonously decreasing regioneven if the viewpoint moves as illustrated in FIG. 15. Therefore, thereliability level of the normal vector can be evaluated properly on thebasis of the relation in magnitude and the difference between thedegrees of polarization.

According to the modes described so far, it is possible to identify,solely from a polarization image, a normal vector and positioncoordinates thereof even in the absence of a feature point such as apattern on the subject surface. Meanwhile, if a location where a featurepoint exists is selected as a point of interest, the position where thepoint of interest in question appears in the captured image of eachviewpoint can be identified, thus simplifying the processes. That is, asillustrated in FIG. 13, there is no need to search for a pixel ofinterest on the straight line obtained by projecting the first viewpointline-of-sight vector, thus allowing for an accurate pixel of interest tobe identified from a feature point silhouette. Therefore, by obtaining,of the second viewpoint captured image, an incident plane relative tothe pixel of interest in question, it is possible to obtain a normalvector from the line of intersection between the incident planes of thefirst and second viewpoints.

Similarly, it is possible to obtain position coordinates of the point ofinterest in the world coordinate system from the intersection betweenthe first and second line-of-sight vectors. In this mode, there is noneed to search for a pixel of interest on the basis of the reliabilitylevel of the normal vector as described above. Therefore, it is alsopossible to obtain position coordinates of the point of interestdirectly without obtaining a normal vector.

In the modes described so far, the angle of the incident plane at eachviewpoint is obtained basically from the phase angle ψ in Formula 1.Meanwhile, incident planes may be obtained by use of the fact that thedegree of polarization depends only upon the zenith angle θ asillustrated in FIG. 14. FIG. 17 is a diagram for describing a techniquefor identifying an incident plane on the basis of the change in thedegree of polarization relative to the zenith angle θ. As describedabove, a silhouette of the point of interest ‘a’ on the subject 72 isobvious even when the viewpoint of the imaging apparatus 12 changes.

Here, in order to determine an incident plane 142 a of the firstviewpoint, a search is made of the direction where a rate of change of azenith angle θa is the largest. That is, the position and posture of theimaging apparatus 12 a are changed in various ways to acquire thedirection where the variation in degree of polarization relative to thechange in unit angle of the line-of-sight vector is the largest. Theplane including the direction of movement of the viewpoint and the pointof interest ‘a’ is the incident plane 142 a. In other words, theincident plane 142 a remains unchanged even if the viewpoint changes inthe direction of movement in question. Therefore, it is only necessaryto select a point in the direction of movement in question as the “firstviewpoint” in this case. As for the second viewpoint, the imagingapparatus 12 b is similarly changed in various ways, and the planeincluding the direction of movement of the viewpoint and the point ofinterest ‘a’ when the rate of variation in degree of polarization is thelargest is the incident plane 142 b.

In this case, although it is necessary to move the imaging apparatus 12in a two-dimensional direction such as horizontally or vertically, theviewpoint control section 60 need only guide the viewpoint bypresenting, to the user, a direction of movement by image or sound asillustrated in FIG. 11. It should be noted that an incident planeacquired by Formula 1 may be integrated with a result of an incidentplane obtained from the rate of change of the degree of polarization.For example, the mean of the two may be used as a true incident planeangle. Also, a different technique may be used to derive each incidentplane. For example, an incident plane of the first viewpoint may beobtained from the phase angle ψ, and incident planes of other viewpointsmay be obtained from the rate of change of the degree of polarization.

Also, in the modes described so far, a specular reflection model isapplied by selecting a pixel of interest having a degree of polarizationequal to or higher than a threshold as a target. Meanwhile, the use ofthe difference in behavior of the degree of polarization relative to thechange in the zenith angle θ makes it possible to apply the presentembodiment also to light where diffuse reflection is predominant. Forexample, in the case where, when the viewpoint is moved such that thezenith angle θ changes, the rate of change of the degree of polarizationdoes not become a given value or higher no matter in which direction theviewpoint is moved, it is probable that light where diffuse reflectionis predominant is being observed.

The reason for this is that the rate of change of the degree ofpolarization is higher with specular reflection in an almost entirerange of the zenith angle as illustrated in FIG. 4. In such a case, anincident plane is derived by applying a diffuse reflection model. Thatis, we assume that there is an incident plane at the angle indicated bythe phase angle ψ in Formula 1. Any other processes are similar to thosedescribed above regarding specular reflection.

According to the present embodiment described above, it is possible tointroduce a specular reflection model for subsequent computations byselecting a pixel of interest having a degree of polarization equal toor higher than a threshold as a target to be processed in image analysisbased on a polarization image. Also, a pixel with a large amplituderelative to a polarization luminance azimuth is selected, thus allowingfor a phase angle to be obtained with high accuracy. As a result, theincident plane angle for the point of interest on the subjectrepresented by the pixel of interest in question can be acquired withhigh accuracy.

Also, the viewpoint is moved in an appropriate direction with referenceto an incident plane acquired for a certain viewpoint, and the samepoint of interest on the subject is captured, thus acquiring a pluralityof incident planes and acquiring, in the world coordinate system, anormal vector of the point of interest from a line of intersection ofthe incident planes. The direction of movement of the viewpoint can bedetermined in such a manner as to ensure accuracy in normal relative tothe first viewpoint, a reference, thus making it possible to obtainnormals efficiently with a small number of image captures. Also, thedirection of movement is clear, thus allowing for image capture toreadily be completed without much time and effort even when the usermanually moves the viewpoint by giving simple instructions by image orsound.

Also, the line-of-sight vector from the first viewpoint toward the pointof interest is projected onto the image plane of the second viewpoint,and a search is made, on the straight line, of a point where a highlyreliable normal vector can be acquired. Further, the positioncoordinates of the point of interest in the world coordinate system areacquired from the intersection between the line-of-sight vector from thesecond viewpoint to the position in question and the line-of-sightvector from the first viewpoint to the point of interest. This makes itpossible to acquire subject state information in the world coordinatesystem even in the absence of a feature point at the point of intereston the subject or even if the position of the point of interest inquestion is unknown.

The present embodiment as a whole uses the relation between the zenithangle and the degree of polarization of a normal vector of the subject,thus allowing subject information to be acquired with high robustness inthe face of an apparent change in luminance attributable to brightnessof the imaging environment, the viewpoint position, a shadow, and thelike. Also, detailed information regarding regions with a high degree ofpolarization, i.e., regions close to a Brewster's angle where areflectance of p polarized light is 0 is acquired, providing excellentaffinity, for example, to the technique disclosed in NPL 1 that carriesout image analysis by using the Brewster's angle as a parameter andmaking application to further image analysis possible.

The present invention has been described above on the basis of anembodiment. The above embodiment is illustrative, and it is understoodby a person skilled in the art that the combination of constituentelements and processes thereof can be modified in various ways and thatsuch modification examples also fall within the scope of the presentinvention.

REFERENCE SIGNS LIST

10 Information processing apparatus, 12 Imaging apparatus, 16 Displayapparatus, 23 CPU, 24 GPU, 26 Main memory, 50 Captured image acquisitionsection, 52 Image data storage section, 53 Imaging apparatus informationacquisition section, 54 Subject information acquisition section, 56Output data generation section, 60 Viewpoint control section, 64Point-of-interest information acquisition section

INDUSTRIAL APPLICABILITY

As described above, the present invention is applicable to various typesof information processing apparatuses such as a gaming apparatus, amobile terminal, a monitoring camera system, a vehicle-mounted camerasystem, and an inspection apparatus.

The invention claimed is:
 1. An information processing apparatuscomprising: a captured image acquisition section adapted to acquire dataof polarization images in a plurality of azimuths captured by an imagingapparatus from different viewpoints; an imaging apparatus informationacquisition section adapted to acquire information regarding a positionand posture of the imaging apparatus as viewpoint information; apoint-of-interest information acquisition section adapted to acquire, byusing polarization luminance of a pixel of interest representing a pointof interest on a subject, an incident plane of observed light at thepoint of interest for each of the viewpoints first, acquirepoint-of-interest state information in a world coordinate system byintegrating the incident planes on a basis of a positional relationbetween the viewpoints, and output the point-of-interest stateinformation; and a viewpoint control section adapted to determine, on abasis of the incident plane acquired relative to a first viewpoint, adirection in which to move the viewpoint and determine a post-movementviewpoint when a given condition is met relative to the incident planeas a second viewpoint for which to derive the incident plane next. 2.The information processing apparatus of claim 1, wherein the viewpointcontrol section guides the viewpoint by displaying the direction inwhich to move the viewpoint on a display apparatus as an image andinstructing a user to move the imaging apparatus according to the image.3. The information processing apparatus of claim 2, wherein theviewpoint control section causes a silhouette within a field of view ofthe imaging apparatus to be displayed on the display apparatus andcauses graphics indicating the direction in which to move the viewpointto be displayed in a superimposed manner.
 4. The information processingapparatus of claim 2, wherein the viewpoint control section detects afeature point of the subject existing in the direction in which to movethe viewpoint for the pixel of interest in the image captured from thefirst viewpoint and guides the viewpoint to move in a direction towardthe feature point.
 5. The information processing apparatus of claim 2,wherein the viewpoint control section instructs the user to attach amarking to a position located at a given spacing in the direction inwhich to move the viewpoint for the pixel of interest in the imagecaptured from the first viewpoint and guides the viewpoint to move in adirection toward the marking.
 6. The information processing apparatus ofclaim 2, wherein the viewpoint control section shines light from a lightbeam shining device onto a position located at a given spacing in thedirection in which to move the viewpoint for the pixel of interest inthe image captured from the first viewpoint and guides the viewpoint tomove in a direction toward a pattern onto which light is shone.
 7. Theinformation processing apparatus of claim 1, wherein thepoint-of-interest information acquisition section acquires a line ofintersection between the incident planes, each acquired for one of theviewpoints, as a normal vector of the point of interest.
 8. Theinformation processing apparatus of claim 1, wherein thepoint-of-interest information acquisition section acquires anintersection between line-of-sight vectors from each viewpoint to thepoint of interest as position coordinates of the point of interest. 9.The information processing apparatus of claim 1, wherein the capturedimage acquisition section acquires video data captured while theviewpoint is moved at the same time and including, in each frame,polarization information in a plurality of azimuths, and the viewpointcontrol section determines a frame acquired from the second viewpoint asa polarization image used to acquire the incident plane.
 10. Theinformation processing apparatus of claim 1, wherein the captured imageacquisition section acquires polarization image data of still imageseach captured from a different viewpoint, and the viewpoint controlsection instructs a user to operate a shutter when the post-movementviewpoint meets the condition.
 11. The information processing apparatusof claim 1, wherein the point-of-interest information acquisitionsection acquires a plane including a direction of movement of theviewpoint that provides a maximum rate of change of a degree ofpolarization of the pixel of interest and the point of interest when theviewpoint is moved as the incident plane having one of points in thedirection of movement, and the viewpoint control section furtherdetermines the direction in which to move the viewpoint so as to acquirethe incident plane.
 12. The information processing apparatus of claim 1,wherein the viewpoint control section determines whether or not thepost-movement viewpoint meets the condition on a basis of a distancefrom the incident plane or an angle relative to the incident planeacquired for the first viewpoint.
 13. A subject information acquisitionmethod by an information processing apparatus, comprising: acquiringdata of polarization images in a plurality of azimuths captured by animaging apparatus from different viewpoints; acquiring informationregarding a position and posture of the imaging apparatus as viewpointinformation; acquiring, by using polarization luminance of a pixel ofinterest representing a point of interest on a subject, an incidentplane of observed light at the point of interest for each of theviewpoints first and acquiring point-of-interest state information in aworld coordinate system by integrating the incident planes on a basis ofa positional relation between the viewpoints; determining, on a basis ofthe incident plane acquired relative to a first viewpoint, a directionin which to move the viewpoint and determining a post-movement viewpointwhen a given condition is met relative to the incident plane as a secondviewpoint for which to derive the incident plane next; and outputtingthe point-of-interest state information.
 14. A non-transitory, computerreadable storage medium containing a computer program, which whenexecuted by a computer, causes the computer to carry out actions,comprising: acquiring data of polarization images in a plurality ofazimuths captured by an imaging apparatus from different viewpoints;acquiring information regarding a position and posture of the imagingapparatus as viewpoint information; acquiring, by using polarizationluminance of a pixel of interest representing a point of interest on asubject, an incident plane of observed light at the point of interestfor each of the viewpoints first and acquiring point-of-interest stateinformation in a world coordinate system by integrating the incidentplanes on a basis of a positional relation between the viewpoints;determining, on a basis of the incident plane acquired relative to afirst viewpoint, a direction in which to move the viewpoint anddetermining a post-movement viewpoint when a given condition is metrelative to the incident plane as a second viewpoint for which to derivethe incident plane next; and outputting the point-of-interest stateinformation.